AITopics | tokenization tutorial

Collaborating Authors

tokenization tutorial

Information about AI from the News, Publications, and Conferences

Automatic Classification – Tagging and Summarization – Customizable Filtering and Analysis

If you are looking for an answer to the question What is Artificial Intelligence? and you only have a minute, then here's the definition the Association for the Advancement of Artificial Intelligence offers on its home page: "the scientific understanding of the mechanisms underlying thought and intelligent behavior and their embodiment in machines."

However, if you are fortunate enough to have more than a minute, then please get ready to embark upon an exciting journey exploring AI (but beware, it could last a lifetime) …

Data Science 101 (Getting started in NLP): Tokenization tutorial

@machinelearnbotSep-14-2017, 15:20:11 GMT

One common task in NLP (Natural Language Processing) is tokenization. "Tokens" are usually individual words (at least in languages like English) and "tokenization" is taking a text or set of text and breaking it up into its individual words. These tokens are then used as the input for other types of analysis or tasks, like parsing (automatically tagging the syntactic relationship between words). In this tutorial you'll learn how to: For this tutorial we'll be using a corpus of transcribed speech from bilingual children speaking in English. You can find more information on this dataset and download it here.

artificial intelligence, natural language, tokenization tutorial, (8 more...)

@machinelearnbot

Technology: Information Technology > Artificial Intelligence > Natural Language (1.00)

Add feedback